Perceiving depth from texture and disparity cues: Evidence for a non-probabilistic account of cue integration.

Bayesian inference theories have been extensively used to model how the brain derives three-dimensional (3D) information from ambiguous visual input. In particular, the maximum likelihood estimation (MLE) model combines estimates from multiple depth cues according to their relative reliability to produce the most probable 3D interpretation. Here, we tested an alternative theory of cue integration, termed the intrinsic constraint (IC) theory, which postulates that the visual system derives the most stable, not most probable, interpretation of the visual input amid variations in viewing conditions. The vector sum model provides a normative approach for achieving this goal where individual cue estimates are components of a multidimensional vector whose norm determines the combined estimate. Individual cue estimates are not accurate but related to distal 3D properties through a deterministic mapping. In three experiments, we show that the IC theory can more adeptly account for 3D cue integration than MLE models. In Experiment 1, we show systematic biases in the perception of depth from texture and depth from binocular disparity. Critically, we demonstrate that the vector sum model predicts an increase in perceived depth when these cues are combined. In Experiment 2, we illustrate the IC theory radical reinterpretation of the just noticeable difference (JND) and test the related vector sum model prediction of the classic finding of smaller JNDs for combined-cue versus single-cue stimuli. In Experiment 3, we confirm the vector sum prediction that biases found in cue integration experiments cannot be attributed to flatness cues, as the MLE model predicts.

The case against probabilistic inference: a new deterministic theory of 3D visual processing.

How the brain derives 3D information from inherently ambiguous visual input remains the fundamental question of human vision. The past two decades of research have addressed this question as a problem of probabilistic inference, the dominant model being maximum-likelihood estimation (MLE). This model assumes that independent depth-cue modules derive noisy but statistically accurate estimates of 3D scene parameters that are combined through a weighted average. Cue weights are adjusted based on the system representation of each module's output variability. Here I demonstrate that the MLE model fails to account for important psychophysical findings and, importantly, misinterprets the just noticeable difference, a hallmark measure of stimulus discriminability, to be an estimate of perceptual uncertainty. I propose a new theory, termed Intrinsic Constraint, which postulates that the visual system does not derive the most probable interpretation of the visual input, but rather, the most stable interpretation amid variations in viewing conditions. This goal is achieved with the Vector Sum model, which represents individual cue estimates as components of a multi-dimensional vector whose norm determines the combined output. This model accounts for the psychophysical findings cited in support of MLE, while predicting existing and new findings that contradict the MLE model. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

New Approaches to 3D Vision.

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Explicit and implicit depth-cue integration: Evidence of systematic biases with real objects.

In previous studies using VR, we found evidence that 3D shape estimation agrees to a superadditivity rule of depth-cue combination, by which adding depth cues leads to greater perceived depth and, in principle, to depth overestimation. Superadditivity can be quantitatively accounted for by a normative theory of cue integration, via adapting a model termed Intrinsic Constraint (IC). As for its qualitative nature, it remains unclear whether superadditivity represents the genuine readout of depth-cue integration, as predicted by IC, or alternatively a byproduct of artificial virtual displays, because they carry flatness cues that can bias depth estimates in a Bayesian fashion, or even just a way for observers to express that a scene "looks deeper" with more depth cues by explicitly inflating their depth judgments. In the present study, we addressed this question by testing whether the IC model's prediction of superadditivity generalizes to real world settings. We asked participants to judge the perceived 3D shape of cardboard prisms through a matching task. To control for the potential interference of explicit reasoning, we also asked participants to reach-to-grasp the same objects and we analyzed the in-flight grip size throughout the reaching. We designed a novel technique to carefully control binocular and monocular 3D cues independently, allowing to add or remove depth information seamlessly. Even with real objects, participants exhibited a clear superadditivity effect in both tasks. Furthermore, the magnitude of this effect was accurately predicted by the IC model. These results confirm that superadditivity is an inherent feature of depth estimation.

Taking aim at the perceptual side of motor learning: exploring how explicit and implicit learning encode perceptual error information through depth vision.

Motor learning in visuomotor adaptation tasks results from both explicit and implicit processes, each responding differently to an error signal. Although the motor output side of these processes has been extensively studied, the visual input side is relatively unknown. We investigated if and how depth perception affects the computation of error information by explicit and implicit motor learning. Two groups of participants made reaching movements to bring a virtual cursor to a target in the frontoparallel plane. The Delayed group was allowed to reaim and their feedback was delayed to emphasize explicit learning, whereas the camped group received task-irrelevant clamped cursor feedback and continued to aim straight at the target to emphasize implicit adaptation. Both groups played this game in a highly detailed virtual environment (depth condition), leveraging a cover task of playing darts in a virtual tavern, and in an empty environment (no-depth condition). The delayed group showed an increase in error sensitivity under depth relative to no-depth. In contrast, the clamped group adapted to the same degree under both conditions. The movement kinematics of the delayed participants also changed under the depth condition, consistent with the target appearing more distant, unlike the Clamped group. A comparison of the delayed behavioral data with a perceptual task from the same individuals showed that the greater reaiming in the depth condition was consistent with an increase in the scaling of the error distance and size. These findings suggest that explicit and implicit learning processes may rely on different sources of perceptual information.NEW & NOTEWORTHY We leveraged a classic sensorimotor adaptation task to perform a first systematic assessment of the role of perceptual cues in the estimation of an error signal in the 3-D space during motor learning. We crossed two conditions presenting different amounts of depth information, with two manipulations emphasizing explicit and implicit learning processes. Explicit learning responded to the visual conditions, consistent with perceptual reports, whereas implicit learning appeared to be independent of them.

Newtonian Predictions Are Integrated With Sensory Information in 3D Motion Perception.

Because the motions of everyday objects obey Newtonian mechanics, perhaps these laws or approximations thereof are internalized by the brain to facilitate motion perception. Shepard's seminal investigations of this hypothesis demonstrated that the visual system fills in missing information in a manner consistent with kinematic constraints. Here, we show that perception relies on internalized regularities not only when filling in missing information but also when available motion information is inconsistent with the expected outcome of a physical event. When healthy adult participants (Ns = 11, 11, 12, respectively, in Experiments 1, 2, and 3) viewed 3D billiard-ball collisions demonstrating varying degrees of consistency with Newtonian mechanics, their perceptual judgments of postcollision trajectories were biased toward the Newtonian outcome. These results were consistent with a maximum-likelihood model of sensory integration in which perceived target motion following a collision is a reliability-weighted average of a sensory estimate and an internal prediction consistent with Newtonian mechanics.

Persistent grasping errors produce depth cue reweighting in perception.

When a grasped object is larger or smaller than expected, haptic feedback automatically recalibrates motor planning. Intriguingly, haptic feedback can also affect 3D shape perception through a process called depth cue reweighting. Although signatures of cue reweighting also appear in motor behavior, it is unclear whether this motor reweighting is the result of upstream perceptual reweighting, or a separate process. We propose that perceptual reweighting is directly related to motor control; in particular, that it is caused by persistent, systematic movement errors that cannot be resolved by motor recalibration alone. In Experiment 1, we inversely varied texture and stereo cues to create a set of depth-metamer objects: when texture specified a deep object, stereo specified a shallow object, and vice versa, such that all objects appeared equally deep. The stereo-texture pairings that produced this perceptual metamerism were determined for each participant in a matching task (Pre-test). Next, participants repeatedly grasped these depth metamers, receiving haptic feedback that was positively correlated with one cue and negatively correlated with the other, resulting in persistent movement errors. Finally, participants repeated the perceptual matching task (Post-test). In the condition where haptic feedback reinforced the texture cue, perceptual changes were correlated with changes in grasping performance across individuals, demonstrating a link between perceptual reweighting and improved motor control. Experiment 2 showed that cue reweighting does not occur when movement errors are rapidly corrected by standard motor adaptation. These findings suggest a mutual dependency between perception and action, with perception directly guiding action, and actions producing error signals that drive motor and perceptual learning.

Sensorimotor adaptation and cue reweighting compensate for distorted 3D shape information, accounting for paradoxical perception-action dissociations.

Visually guided movements can show surprising accuracy even when the perceived three-dimensional (3D) shape of the target is distorted. One explanation of this paradox is that an evolutionarily specialized "vision-for-action" system provides accurate shape estimates by relying selectively on stereo information and ignoring less reliable sources of shape information like texture and shading. However, the key support for this hypothesis has come from studies that analyze average behavior across many visuomotor interactions where available sensory feedback reinforces stereo information. The present study, which carefully accounts for the effects of feedback, shows that visuomotor interactions with slanted surfaces are actually planned using the same cue-combination function as slant perception and that apparent dissociations can arise due to two distinct supervised learning processes: sensorimotor adaptation and cue reweighting. In two experiments, we show that when a distorted slant cue biases perception (e.g., surfaces appear flattened by a fixed amount), sensorimotor adaptation rapidly adjusts the planned grip orientation to compensate for this constant error. However, when the distorted slant cue is unreliable, leading to variable errors across a set of objects (i.e., some slants are overestimated, others underestimated), then relative cue weights are gradually adjusted to reduce the misleading effect of the unreliable cue, consistent with previous perceptual studies of cue reweighting. The speed and flexibility of these two forms of learning provide an alternative explanation of why perception and action are sometimes found to be dissociated in experiments where some 3D shape cues are consistent with sensory feedback while others are faulty.NEW & NOTEWORTHY When interacting with three-dimensional (3D) objects, sensory feedback is available that could improve future performance via supervised learning. Here we confirm that natural visuomotor interactions lead to sensorimotor adaptation and cue reweighting, two distinct learning processes uniquely suited to resolve errors caused by biased and noisy 3D shape cues. These findings explain why perception and action are often found to be dissociated in experiments where some cues are consistent with sensory feedback while others are faulty.

Depth cue reweighting requires altered correlations with haptic feedback.

Depth cue reweighting is a feedback-driven learning process that modifies the relative influences of different sources of three-dimensional shape information in perceptual judgments and or motor planning. In this study, we investigated the mechanism supporting reweighting of stereo and texture information by manipulating the haptic feedback obtained during a series of grasping movements. At the end of each grasp, the fingers closed down on a physical object that was consistent with one of the two cues, depending on the condition. Previous studies have shown that this style of visuomotor training leads to cue reweighting for perceptual judgments, but the time course has never been documented for a single training session, and many questions remain regarding the underlying mechanism, such as the pattern of feedback signals required to drive reweighting. We address these issues in two experiments, finding short-term changes in the motor response consistent with cue reweighting: the slope of the grip aperture with respect to the reliable cue increased, whereas the slope with respect to the unreliable cue decreased. Critically, Experiment 2 shows that slope changes do not occur when one of the cues is rendered with a constant bias; the grip aperture simply becomes uniformly larger or smaller. Our findings support a model of cue reweighting driven by altered correlations between haptic feedback and individual cues, rather than simple mismatches, which can be resolved by other mechanisms such as sensorimotor adaptation or cue recalibration.

Does depth-cue combination yield identical biases in perception and grasping?

Grasping critically depends on stereo information. We previously found that binocular disparities yield a distorted visual space, in which objects close to the observer are grasped and perceived as if they were more elongated than farther objects. Such lack of shape constancy results from the inaccurate estimate of the viewing distance, which affects the estimated depth-to-width ratio of an object. This is because (1) depth from binocular disparities scales with the square of the distance and (2) width from retinal size scales linearly with distance. Conversely, depth from monocular cues (i.e., motion and texture gradients) scales linearly with distance, hence the overall shape from these signals should not be affected by errors in egocentric estimates of object location. We therefore reasoned that adding these cues to stereo information should improve shape constancy. Contrary to expectations, in four experiments we found that stereo-texture and stereo-motion stimuli appeared even more distorted than stereo stimuli. More remarkably, results revealed that grasping execution showed identical biases, which were corrected only late in the movement through online control mechanisms, but only if both grasping digits could be visually guided on their respective contact locations. On the contrary, when the index was occluded by the object, biases in shape estimation continued to affect grasping execution until movement completion. Moreover, while the initial part of the grasp showed evidence of collision avoidance, a control experiment suggested that the above biases could have emerged as early as at movement planning, consistent with previous evidence. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Multiple distance cues do not prevent systematic biases in reach to grasp movements.

The perceived distance of objects is biased depending on the distance from the observer at which objects are presented, such that the egocentric distance tends to be overestimated for closer objects, but underestimated for objects further away. This leads to the perceived depth of an object (i.e., the perceived distance from the front to the back of the object) also being biased, decreasing with object distance. Several studies have found the same pattern of biases in grasping tasks. However, in most of those studies, object distance and depth were solely specified by ocular vergence and binocular disparities. Here we asked whether grasping objects viewed from above would eliminate distance-dependent depth biases, since this vantage point introduces additional information about the object's distance, given by the vertical gaze angle, and its depth, given by contour information. Participants grasped objects presented at different distances (1) at eye-height and (2) 130 mm below eye-height, along their depth axes. In both cases, grip aperture was systematically biased by the object distance along most of the trajectory. The same bias was found whether the objects were seen in isolation or above a ground plane to provide additional depth cues. In two additional experiments, we verified that a consistent bias occurs in a perceptual task. These findings suggest that grasping actions are not immune to biases typically found in perceptual tasks, even when additional cues are available. However, online visual control can counteract these biases when direct vision of both digits and final contact points is available.

The endless visuomotor calibration of reach-to-grasp actions.

It is reasonable to assume that when we grasp an object we carry out the movement based only on the currently available sensory information. Unfortunately, our senses are often prone to err. Here, we show that the visuomotor system exploits the mismatch between the predicted and sensory outcomes of the immediately preceding action (sensory prediction error) to attain a degree of robustness against the fallibility of our perceptual processes. Participants performed reach-to-grasp movements toward objects presented at eye level at various distances. Grip aperture was affected by the object distance, even though both visual feedback of the hand and haptic feedback were provided. Crucially, grip aperture as well as the trajectory of the hand were systematically influenced also by the immediately preceding action. These results are well predicted by a model that modifies an internal state of the visuomotor system by adjusting the visuomotor mapping based on the sensory prediction errors. In sum, the visuomotor system appears to be in a constant fine-tuning process which makes the generation and control of grasping movements more resistant to interferences caused by our perceptual errors.

Transfer of adaptation reveals shared mechanism in grasping and manual estimation.

An influential idea in cognitive neuroscience is that perception and action are highly separable brain functions, implemented in distinct neural systems. In particular, this theory predicts that the functional distinction between grasping, a skilled action, and manual estimation, a type of perceptual report, should be mirrored by a split between their respective control systems. This idea has received support from a variety of dissociations, yet many of these findings have been criticized for failing to pinpoint the source of the dissociation. In this study, we devised a novel approach to this question, first targeting specific grasp control mechanisms through visuomotor adaptation, then testing whether adapted mechanisms were also involved in manual estimation - a response widely characterized as perceptual in function. Participants grasped objects in virtual reality that could appear larger or smaller than the actual physical sizes felt at the end of each grasp. After brief exposure to a size perturbation, manual estimates were biased in the same direction as the maximum grip apertures of grasping movements, indicating that the adapted mechanism is active in both tasks, regardless of the perception-action distinction. Additional experiments showed that the transfer effect generalizes broadly over space (Exp. 1B) and does not appear to arise from a change in visual perception (Exp. 2). We discuss two adaptable mechanisms that could have mediated the observed effect: (a) an afferent proprioceptive mechanism for sensing grip shape; and (b) an efferent visuomotor transformation of size information into a grip-shaping motor command.

On the response function and range dependence of manual estimation.

Manual estimates without vision of the hand are thought to constitute a form of cross-modal matching between stimulus size and finger opening. However, few investigations have systematically looked at how manual estimates relate a perceived size to the response across different ranges of stimuli. In two experiments (N = 18 and N = 14), we sought to map out the response properties for (1) manual estimates of visually presented stimuli as well as (2) visual estimates of proprioceptive stimuli, and to test whether these properties depend on the range of stimuli. We also looked at whether scalar variability is present in manual estimates, as predicted by Weber's Law for perceptual tasks. We found that manual estimates scale linearly and with a slope of close to 1 with object sizes up to 90 mm, before participants' hand size limited their responses. In contrast, we found a shallower response slope of about 0.7 when participants performed the inverse task, adjusting the size of a visual object to match a not actively chosen, induced finger opening. Our results were mixed with regards to scalar variability in large objects. We saw some indication of a plateau, but no evidence for an effect of mechanical constraints in the range studied (up to 90 mm). Participants also showed a clear tendency to overestimate small differences when a set of objects differed little in size, but not when stimulus differences were more pronounced.

How removing visual information affects grasping movements.

Our interaction with objects is facilitated by the availability of visual feedback. Here, we investigate how and when visual feedback affects the way we grasp an object. Based on the main views on grasping (reach-and-grasp and double-pointing views), we designed four experiments to test: (1) whether the availability of visual feedback influences the digits independently, and (2) whether the absence of visual feedback affects the initial part of the movement. Our results show that occluding (part of) the hand's movement path influences the movement trajectory from the beginning. Thus, people consider the available feedback when planning their movements. The influence of the visual feedback depends on which digit is occluded, but its effect is not restricted to the occluded digit. Our findings indicate that the control mechanisms are more complex than those suggested by current views on grasping.

Does visuomotor adaptation contribute to illusion-resistant grasping?

Do illusory distortions of perceived object size influence how wide the hand is opened during a grasping movement? Many studies on this question have reported illusion-resistant grasping, but this finding has been contradicted by other studies showing that grasping movements and perceptual judgments are equally susceptible. One largely unexplored explanation for these contradictions is that illusion effects on grasping can be reduced with repeated movements. Using a visuomotor adaptation paradigm, we investigated whether an adaptation model could predict the time course of Ponzo illusion effects on grasping. Participants performed a series of trials in which they viewed a thin wooden target, manually reported an estimate of the target's length, then reached to grasp the target. Manual size estimates (MSEs) were clearly biased by the illusion, but maximum grip apertures (MGAs) of grasping movements were consistently accurate. Illusion-resistant MGAs were observed immediately upon presentation of the illusion, so there was no decrement in susceptibility for the adaptation model to explain. To determine whether online corrections based on visual feedback could have produced illusion-resistant MGAs, we performed an exploratory post hoc analysis of movement trajectories. Early portions of the illusion effect profile evolved as if they were biased by the illusion to the same magnitude as the perceptual responses (MSEs), but this bias was attenuated prior to the MGA. Overall, this preregistered study demonstrated that visuomotor adaptation of grasping is not the primary source of illusion resistance in closed-loop grasping.

Error correction and spatial generalization in human grasp control.

The visual processes that support grasp planning are often studied by analyzing averaged kinematics of repeated movements, as in the literature on grasping and visual illusions. However, by recalibrating visuomotor mappings, the sensorimotor system can adjust motor outputs without changing visual processing, which complicates the interpretation of averaged behavior. We developed a dynamic model of grasp planning and adaptation that can explain why some studies find decrements in illusion effects on grasping while others do not. In two experiments, we tested grasping in a standard three-phase adaptation paradigm and analyzed adaptation aftereffects on the maximum grip aperture as well as the error correction parameters estimated by our model. Experiment 1 demonstrated that the model accounts for adaptive responses to positive and negative visual size perturbations. Experiment 2 supported the novel hypothesis that visuomotor mappings for grasp planning can compensate for opposing size perturbations when these perturbations are experienced in separate regions of space. Our findings serve to illustrate how the surprising flexibility of grasp adaptation can hide (especially in session-wise averages) the true effects of visual perturbations on the visual processes that drive grasp planning.

Stereovision for action reflects our perceptual experience of distance and depth.

Binocular vision is widely recognized as the most reliable source of 3D information within the peripersonal space, where grasping takes place. Since grasping is normally successful, it is often assumed that stereovision for action is accurate. This claim contradicts psychophysical studies showing that observers cannot estimate the 3D properties of an object veridically from binocular information. In two experiments, we compared a front-to-back grasp with a perceptual depth estimation task and found that in both conditions participants consistently relied on the same distorted 3D representation. The subjects experienced (a) compression of egocentric distances: objects looked closer to each other along the z-axis than they were, and (b) underconstancy of relative depth: closer objects looked deeper than farther objects. These biases, which stem from the same mechanism, varied in magnitude across observers, but they equally affected the perceptual and grasping task of each subject. In a third experiment, we found that the visuomotor system compensates for these systematic errors, which are present at planning, through online corrections allowed by visual and haptic feedback of the hand. Furthermore, we hypothesized that the two phenomena would give rise to estimates of the same depth interval that are geometrically inconsistent. Indeed, in a fourth experiment, we show that the landing positions of the grasping digits differ systematically depending on whether they result from absolute distance estimates or relative depth estimates, even when the targeted spatial locations are identical.

Adaptation effects in grasping the Müller-Lyer illusion.

Recent results have shown that effects of pictorial illusions in grasping may decrease over the course of an experiment. This can be explained as an effect of sensorimotor learning if we consider a pictorial size illusion as simply a perturbation of visually perceived size. However, some studies have reported very constant illusion effects over trials. In the present paper, we apply an error-correction model of adaptation to experimental data of N=40 participants grasping the Müller-Lyer illusion. Specifically, participants grasped targets embedded in incremental and decremental Müller-Lyer illusion displays in (1) the same block in pseudo-randomised order, and (2) separate blocks of only one type of illusion each. Consistent with predictions of our model, we found an effect of interference between the two types when they were presented intermixed, explaining why adaptation rates may vary depending on the experimental design. We also systematically varied the number of object sizes per block, which turned out to have no effect on the rate of adaptation. This was also in accordance with our model. We discuss implications for the illusion literature, and lay out how error-correction models can explain perception-action dissociations in some, but not all grasping-of-illusion paradigms in a parsimonious and plausible way, without assuming different illusion effects.

On-line visual control of grasping movements.

Even though it is recognized that vision plays an important role in grasping movements, it is not yet fully understood how the visual feedback of the hand contributes to the on-line control. Visual feedback could be used to shape the posture of the hand and fingers, to adjust the trajectory of the moving hand, or a combination of both. Here, we used a dynamic perturbation method that altered the position of the visual feedback relative to the actual position of the thumb and index finger to virtually increase or decrease the visually sensed grip aperture. Subjects grasped objects in a virtual 3D environment with haptic feedback and with visual feedback provided by small virtual spheres anchored to the their unseen fingertips. We found that the effects of the visually perturbed grip aperture arose preeminently late in the movement when the hand was in the object's proximity. The on-line visual feedback assisted both the scaling of the grip aperture to properly conform it to the object's dimension and the transport of the hand to correctly position the digits on the object's surface. However, the extent of these compensatory adjustments was contingent on the viewing geometry. The visual control of the actual grip aperture was mainly observed when the final grasp axis orientation was approximately perpendicular to the viewing direction. On the contrary, when the final grasp axis was aligned with the viewing direction, the visual control was predominantly concerned with the guidance of the digit toward the visible final contact point.